Problem Statement and Metrics
Learn about the problem statement and metrics for building a video recommendation system.
2. Metrics design and requirements#
Metrics#
Offline metrics#
- Use precision, recall, ranking loss, and logloss.
Online metrics#
- Use A/B testing to compare Click Through Rates, watch time, and Conversion rates.
Requirements#
Training#
- User behavior is generally unpredictable, and videos can become viral during the day. Ideally, we want to train many times during the day to capture temporal changes.
Inference#
-
For every user to visit the homepage, the system will have to recommend 100 videos for them. The latency needs to be under 200ms, ideally sub 100ms.
-
For online recommendations, it’s important to find the balance between exploration vs. exploitation. If the model over-exploits historical data, new videos might not get exposed to users. We want to balance between relevancy and fresh new content.
Summary#
| Type | Desired goals |
|---|---|
| Metrics | Reasonable precision, high recall |
| Training | High throughput with the ability to retrain many times per day |
| Inference | Latency from 100ms to 200ms |
| Flexible to control exploration versus exploitation |
Metrics Evaluation
Candidate Generation and Ranking Model
Mark as Completed
Report an Issue
